This invention relates to the field of determining the size of an object from computerized optical image scanning devices. More specifically, the invention is a trainable system and method relating to recognizing the size of bulk items using image processing.
Image processing systems exist in the prior art for estimating the size of objects. Often these systems use binary images to perform this computation. One common method is the use of the zero-th order moment of the binary picture function, or simply put, the number of picture elements that lie within the binary segment.
If tile size and orientation of the object in the image is known in advance, which is the case in inspection problems, for example, binary matched filters are commonly used. This method allows for determining with great precision whether a specific object of certain size and shape is present in the image at a certain location.
The prior art often performs these methods to verify that the target object in the image is indeed the object that is expected, and, possibly, to grade/classify tile object according to the quality of its appearance relative to its zero order moment or matched filter. An alternative purpose could be to identify the target object by matching the target image object with a number of reference matched filters.
In this description, identifying or measuring the size of one or more objects is defined as determining, given a set of reference sizes, the reference size of the target object. Classifying or grading the size is defined as determining that the target object is of a certain class representing a range of sizes and/or that the size or the object is satisfactory Here, one of tile classes can be a xe2x80x9crejectxe2x80x9d class meaning that the size of the target object is not one or the expected Values. Verifying, on the other hand, is defined as determining that the target is known to be a certain size and simply verifying this to be true or false. Recognizing is defined as identifying, measuring, classifying, grading, and/or verifying.
A round object, in this description, is ail object having every part of tile surface or circumference equidistant from the center. Bulk items include any item that is sold in bulk in supermarkets, grocery stores, retail stores or hardware stores.
Examples include produce (fruits and vegetables), sugar, coffee beans, candy, nails, nuts, bolts, general hardware, parts, and package goods.
In image processing, a digital image is an analog image from a camera that is converted to a discrete representation by dividing the picture into a fixed number of locations called picture elements and quantizing the value of the image at those picture elements into a fixed number of values. The resulting digital image can bexe2x80x94processed by a computer algorithm to develop other images or characteristics of these images. These images or characteristics can be stored in memory and/or used to determine information about the imaged object. A pixel is a picture element of a digital image.
Image processing and computer vision is the processing by a computer of a digital image to modify the image or to obtain from the image properties of the imaged objects such as object identity, location, size, etc.
A scene contains one or more objects that are of interest and the surroundings which also get imaged along with the objects. These surroundings are called the background. The background is Usually further away from the camera than the object(s) of interest.
Segmenting (also called figure/ground separation) is separating a scene image into separate object and background images. Segmenting refers to identifying those image pixels that are contained in tile image of the object versus those that belong to tile image of the background. The segmented object image is then tile collection or pixels that comprises the object in tile original image of tile complete scene. The area of a segmented object image is the number of pixels in the object image.
Illumination is the light that illuminates the scene and objects in it. Illumination of tile whole scene directly determines tile illumination of individual objects in the scene and therefore the reflected light of the objects received by imaging apparatus such as video camera.
Ambient illumination is illuminations from any light source except the special lights used specifically for imaging in object. For example, ambient illumination is the illumination due to light sources Occurring in tile environment such as the sun outdoors and room lights indoors.
Glare or specular reflection is the high amount of light reflected off a shiny (specular, exhibiting mirror-like properties -possibly locally) object. The color or the glare is mostly that of the illuminating light (as opposed to tile natural color of the object).
A feature of an image is defined as any property of the image which can be computationally extracted. Features typically have numerical values that can lie in a certain range, say, R0-R1. In prior art, histograms are computed over a whole image or windows (sub-images) in an image. A histogram of q feature of an image is a numerical representation of the distribution of feature values over the image or window. A histogram of a feature is developed by dividing the feature range, R0-R1, into M intervals (bins) and computing the feature for each image pixel. Simply counting how many image or window pixels fall in each bin gives the feature histogram.
Image features include, but are not limited to, features that are related to tile size of the objects in the image. The simplest features related to size of an object are the object pixels. The boundary pixels, subsets of boundary pixels, and characteristics determined from subsets of boundary pixels are also image features related to object size.
U.S. Pat. No. 4,515,275 to Mills and Richert discloses an apparatus and method for processing fruit and the like, particularly for sorting as a function of variables including color, blemish, size and shape. The fruit is moving on a conveyer belt while being rotated and imaged by a line scanning diode array. The line scanning diode array is sufficiently long such that the scanning line is longer than the fruit item and gives information about the length of the fruit. The number of individual detector signals which reflect presence of the fruit contains information to determine the width of the fruit. These numbers are squared and summed, tile result being a representation or fruit volume, a characteristic related to fruit size.
U.S. Pat. No. 5,020,675 to Cowlin et al. discloses ail apparatus for sorting conveyed articles. Sorting of food products Such as vegetables or fruit, is achieved in ac- with their size, weight and color, or the presence of defects oil them. Size is determined by the combination of examining tile leading and following trailing edge of a trace and the color count of each article oil the conveyer. To this information, weight information from load cells can be added.
The use of a zero order moment of a binary thresholded image of ail object is an effective method for identifying tile size of ail object in an image. Similarly, the use of matched binary filters is effective for verifying the size of a target object in tile image. I-lie use of multiple matched binary filters allows for classifying the size of ail object. The reason is that under well controlled imaging conditions, good segmentations can be obtained which, in turn, allow for precise measure-with the above methods.
Both for moment and matching techniques to work for object size recognition, very precise segmentations of the object from the background are needed. Fur- for matched filtering techniques, the exact orientation of the object in the image has to be known a priori.
In summary, much of the prior art in the agricultural arena is concerned with classifying/grading produce items. This prior art can only classify/identify objects/products/produce if they pass a scanner one object at a time. It is also required that the range of sizes (from smallest to largest possible object size) of the object/product/produce be known beforehand. These systems will fail if more than one item is scanned at the same time, or to be more precise, if more than one object appears at a scanning position at the same time. That is, the objects have to be singulated before size recognition.
Further, the prior art often requires a carefully engineered and expensive mechanical environment with carefully controlled lighting conditions where the items are transported to predefined spatial locations. This does not allow for the prior art to distinguish between produce sizes in a cost effective fashion. Prior art will fail if there are multiple instances, possibly touching and overlapping, of the object present at the time scanning takes place. Prior art size recognition/grading sys- also do not allow for the objects to be packaged in transparent plastic bags. Mechanical means of transporting the objects and algorithms for segmenting object images from background images will fail under these conditions.
Additionally, none of the prior art are trainable systems where, through human or computer intervention, new item sizes are learned or old item sizes discarded. That is, the systems can not be taught to recognize object sizes that were not originally programmed in the system or stop recognizing object sizes that were originally programmed in the system.
One area where the prior art has failed to be effective is in produce check out. The current means and methods for checking out produce poses problems Affixing (PLUxe2x80x94price lookup) labels to fresh produce is disliked by customers and produce retailers/wholesalers. Pre-packaged produce items are disliked, because of increased cost of packaging, disposal (solid waste), and inability to inspect produce quality in pre-packaged form.
The process of produce check-out has not changed much since tile first appear- of grocery stores. At the point of sale (POS), the cashier has to recognize the produce item, weigh or Count tile items(s), and determine the price. Currently, in most stores the latter is achieved by manually entering the non-mnemonic PLU code that is associated with the produce. These codes are available at the POS in the form of printed list or in a booklet with pictures.
Problems arise from this process of produce check-out. Certain produce items, e.g., apples and oranges, are graded and priced by variety and size. Size is described in terms of the number of apples contained in a box: 48s are extremely large, with only 48 filling a box, and 216s are the smallest that are commercially packed, with 216 fitting in the same size box. It is difficult for a cashier to visually estimate the size of certain variety of produce since no comparative example is available at the POS.
An object of this invention is an improved apparatus and method for recognizing the size of objects such as produce.
Another object of this invention is an improved trainable apparatus and method for recognizing the size of objects such as produce.
Another object of this invention is an improved apparatus and method for recognizing and pricing objects such as produce based on size at the point of sale, or in the produce department.
The present invention is a system and apparatus that uses image processing to recognize or measure the size of objects within a scene. The system includes all illumination source for illuminating the scene. By controlling the illumination source, all image processing system can take a first digitized image of the scene with the object illuminated at a higher level and a second digitized image with the object illuminated at a lower level. Using all algorithm, the object(s) image is novelly segmented from a background image Of the Scene by a comparison of the two digitized images taken. A processed image (that can be used to characterize object size features) of the round object(s) is then compared to stored reference images. The size of the object is recognized when a match occurs.
Processed images of an object(s) of unrecognized size can be labeled with the actual size of the object and stored in memory, based on certain criteria, so that the size of an object will be recognized when it is imaged in the future. In this novel way, the invention is taught to recognize the size of previously unknown objects.
Size recognition is independent of the number of the round objects because a plurality of size measurements is determined from the object""s boundaries and the object size is novelly determined from this plurality of size measurements. In this way, the invention is taught to recognize the size of an object of previously unknown size.
A user interface and apparatus that determines other features of the object (like color, texture) for identification of the object can be used with the system.